Deeper neural networks are hard to train. Inspired by the elastic collision model in physics, we present a universal structure that could be integrated into the existing network structures to speed up the training process and eventually increase its generalization ability. We apply our structure to the Con-volutional Neural Networks(CNNs) to form a new structure, which we term the "Interlayer Collision" (IC) structure. The IC structure provides the deeper layer a better representation of the input features. We evaluate the IC structure on CI-FAR10 and Imagenet by integrating it into the existing state-of-the-art CNNs. Our experiment shows that the proposed IC structure can effectively increase the accuracy and convergence speed.